Multi-Sensor Analysis of Food

ABSTRACT

In an embodiment, a method for estimating a composition of food includes: receiving a first three-dimensional (3D) image; identifying food in the first 3D image; determining a volume of the identified food based on the first 3D image; and estimating a composition of the identified food using a millimeter-wave radar.

This application is a continuation of U.S. patent application Ser. No.17/368,570, filed Jul. 6, 2021, which application claims the benefit ofU.S. Provisional Application No. 63/049,823, entitled “Multi-SensorAnalysis of Food,” and filed on Jul. 9, 2020, which application ishereby incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates generally to an electronic system andmethod, and, in particular embodiments, to a multi-sensor analysis offood.

BACKGROUND

Diet and exercise are two of the most important factors in our overallhealth. For example, poor diet and exercise habits may increase the riskof obesity, heart disease, cancer, diabetes, and other chronic healthproblems.

Technology companies have tackled the exercise space vigorously,producing a wide range of products, such as connected spinning bikes,step counters, and smartphone apps with guided programs. Diettechnology, however, has remained primitive in comparison. Discountingpharmaceutical and surgical solutions for weight loss, diet technologyis basically limited to smartphone apps. These apps range from foodloggers to platforms that connect dietitians/nutritionists with users tovirtual commercial weight loss programs.

Conventional tools and methods for tracking food intake have been mainlylimited to manual food logging or camera-based image analysis. Forexample, FIG. 1 shows an exemplary food logging app.

Manual food logging is generally time-consuming, inconvenient, andcumbersome. For example, in some cases, a user must log every morsel offood that the user consumes consume. In some cases, this requiressearching for each ingredient, as well as either estimating or weighingthe volume and/or weight of the food consumed. For chain restaurants,the user may be able to find the meal and populate the logautomatically. For prepared foods, the user may be able to scan thebarcode on the package to access the ingredients and nutritionalinformation. In any case, user input is generally needed, and such inputgenerally requires a substantial amount of work.

SUMMARY

In accordance with an embodiment, a method for estimating a compositionof food includes: receiving a first 3D image; identifying food in thefirst 3D image; determining a volume of the identified food based on thefirst 3D image; and estimating a composition of the identified foodusing a millimeter-wave radar.

In accordance with an embodiment, a mobile device includes: a 3D imagingsensor configured to generate a first 3D image; a millimeter-wave radarconfigured to transmit radar signals and receive reflected radarsignals; and a processor configured to: identify food in the first 3Dimage; determine a volume of the identified food based on the first 3Dimage; and estimate a composition of the identified food based on thereflected radar signals.

In accordance with an embodiment, A system including: a time-of-flightsensor; a camera; and a millimeter-wave radar configured to transmitradar signals and receive reflected radar signals; and a processorconfigured to: identify food based on an output of the camera; determinea volume of the identified food based on an output of the time-of-flightsensor; estimate a composition of the identified food based on thereflected radar signals; and determine a quantity of macronutrients ofthe identified food based on the determined volume and the estimatedcomposition of the identified food.

In accordance with an embodiment, a method for estimating a compositionof food includes: receiving radar data from an ADC of a millimeter-waveradar; preprocessing the received radar data to generate a radar image;generating a measurement vector based on the radar image; and estimatinga composition of food based on the measurement vector and a firstsensing matrix, the first sensing matrix including radar signatures of aplurality of types of food.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, and theadvantages thereof, reference is now made to the following descriptionstaken in conjunction with the accompanying drawings, in which:

FIG. 1 shows an exemplary food logging app;

FIG. 2 shows a conventional way of identifying food using a bar code;

FIG. 3 shows an exemplary composition of various chocolate bars;

FIG. 4 shows a diagram with an exemplary composition of foods of ahealthy diet;

FIGS. 5A and 5B show dielectric constants and dielectric loss factor,respectively, of various food materials at 25° C.;

FIG. 6 shows a smartphone, according to an embodiment of the presentinvention;

FIG. 7 shows a flow chart of an embodiment method for classifying food,according to an embodiment of the present invention;

FIG. 8 shows a screen of the smartphone of FIG. 6 while displaying aburger, according to an embodiment of the present invention;

FIGS. 9 and 10 show examples of possible ways to display the compositionof the food estimated while performing the method of FIG. 7 , accordingto embodiments of the present invention;

FIG. 11 shows a schematic diagram of the millimeter-wave radar of FIG. 7, according to an embodiment of the present invention;

FIG. 12 shows a block diagram of an embodiment method for performingfood classification using the millimeter-wave radar of FIG. 11 ,according to an embodiment of the present invention;

FIGS. 13A and 13B illustrate possible embodiments for performing aportion of the method of FIG. 12 , according to embodiments of thepresent invention;

FIG. 14 shows a block diagram of an embodiment method for performingfood classification using the millimeter-wave radar of FIG. 11 ,according to an embodiment of the present invention; and

FIG. 15 illustrates a block diagram showing training and operation of aDCNN used during identification of food components, according to anembodiment of the present invention.

Corresponding numerals and symbols in different figures generally referto corresponding parts unless otherwise indicated. The figures are drawnto clearly illustrate the relevant aspects of the preferred embodimentsand are not necessarily drawn to scale.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The making and using of the embodiments disclosed are discussed indetail below. It should be appreciated, however, that the presentinvention provides many applicable inventive concepts that can beembodied in a wide variety of specific contexts. The specificembodiments discussed are merely illustrative of specific ways to makeand use the invention, and do not limit the scope of the invention.

The description below illustrates the various specific details toprovide an in-depth understanding of several example embodimentsaccording to the description. The embodiments may be obtained withoutone or more of the specific details, or with other methods, components,materials and the like. In other cases, known structures, materials oroperations are not shown or described in detail so as not to obscure thedifferent aspects of the embodiments. References to “an embodiment” inthis description indicate that a particular configuration, structure orfeature described in relation to the embodiment is included in at leastone embodiment. Consequently, phrases such as “in one embodiment” thatmay appear at different points of the present description do notnecessarily refer exactly to the same embodiment. Furthermore, specificformations, structures or features may be combined in any appropriatemanner in one or more embodiments.

Embodiments of the present invention will be described in a specificcontext, a, e.g., 3-dimensional (3D), analysis of food intake, e.g., formobile devices. Embodiments of the present invention may also be usedwith devices other than mobile devices. Some embodiments may perform theanalysis of food without using 3D techniques. It is understood that theterm food includes solid and liquid food, as well as beverages. Someembodiments may be used for analyzing substances other than food.

In an embodiment of the present invention, multiple sensors are fused tosupport a compelling user experience for tracking food intake. A 3Dimaging sensor and a millimeter-wave radar are used to track foodconsumptions. In some embodiments, a 2D RGB camera is also used to,e.g., improve the accuracy of food classification.

Camera-based loggers may be used to identify the food in a picture,e.g., using image recognition, and to estimate the volume of food. Forexample, FIG. 2 shows a conventional way of identifying food using a barcode.

With a 2D camera, volume may be estimated by moving the camera andtaking images at different angles or by placing the food on a tableclothor plate of a known size. Other approaches include weighing the food ona smart plate and using a 3D camera to measure the volume of food.

Camera-based loggers, however, may fail to identify food hidden in theimage and/or fail to convey relevant details about food preparation,such as the amount of oil used, fat content of meats, sugar or saltcontent in baked goods, etc. Therefore, it is not unusual for caloricestimates of conventional camera-based loggers to be 20% off. In somecases, in a 2,000 calorie diet, a 20% error could cause a dieter to gain1 pound every 9 days, where 1 pound is equivalent to 3,500 calories.

In general, calorie-driven food logging may neglect the quality of thefood consumed, from a nutritional standpoint.

In industrial applications, such as in the food industry, hyperspectralimaging may be used to analyze food composition for quality control. Forexample, FIG. 3 shows an exemplary composition of various chocolatebars. Regions high in fat are shown in a first color (e.g., red), whilecrystalline sucrose and moisture are shown in second and third colors(e.g., green and blue), respectively. Combinations of components may beshown as mixed colors. For example, nuts may be shaded with the firstcolor (e.g., red) indicating high-fat content. Caramel may appear in thethird color or a fourth color (e.g., blue or purple) indicating highmoisture content with varying fat content. Chocolate may appear in thesecond color a fifth color (e.g., green, yellow) or a sixth color (e.g.,orange) indicating varying combinations of fat and crystalline sucrose.

Hyperspectral imaging is an expensive process—in both computational(image processing) cost and hardware cost—limiting its commercialadoption thus far to industrial applications.

Recent research indicates that it may be advantageous for people tofocus on the quality of diet, instead of caloric consumption. Currently,estimating macronutrients (e.g., carbohydrates, proteins, and fats) maybe more challenging than counting calories, as it may be tedious toassess the quantity of each food item, look up its macronutrientprofile, and tally up the carbohydrates, proteins, and fats. Aconventional way to approximate a macronutrient profile with littleeffort, thus, is to segment the different types of food that isconsumed. For example, as shown in FIG. 4 , the Harvard School of PublicHealth recommends ½ of your food should be fruits and vegetables, ¼should be proteins, and ¼ should be whole grains.

In an embodiment of the present invention, a 3D imaging sensor, such asincluding a time-of-flight (ToF) sensor, is used to generate a 3D image,which is used to estimate the volume of the food. A millimeter-waveradar is used to estimate the proportion of macronutrients in the food.In some embodiments, the quantity of macronutrients is estimated basedon the estimated proportion of macronutrients and the estimated volumeof the food. In some embodiments, the proportion of macronutrients inthe food is estimated without estimating calories. In some embodiments,focusing on macronutrient composition and avoiding estimating caloriesadvantageously aids in improving the quality of diet since calorieestimation may be prone to error and, e.g., calorie counting may be lesseffective for weight loss and overall health than macronutrientscomposition balancing.

In some embodiments, a millimeter-wave radar is used to distinguish,e.g., between vegetables, fruits, grains, proteins, fats, and beveragetypes. For example, radar reflections are affected by the dielectricproperties of a material, among other factors. Thus, in someembodiments, a millimeter-wave radar is used to distinguish betweentypes of foods (e.g., between vegetables, fruits, grains, proteins,fats, and beverage types) based on signatures of the different radarreflections caused by different dielectric properties of the differentfoods. FIGS. 5A and 5B show dielectric constants and dielectric lossfactor, respectively, of various food materials at 25° C.

In some embodiments, a millimeter-wave radar may be used at differentfrequencies (radar spectroscopy) to analyze food composition andquality. For example, in some embodiments, a millimeter-wave radar isused to estimate the sugar content in a yogurt or beverage.

In some embodiments, a 2D RGB camera may be used to augment the system'saccuracy using image recognition. For example, in some embodiments,image recognition provides a secondary source of data to fuse with theradar data in food type recognition. In some embodiments, imagerecognition may be used for separating different piles of food in theimages, as well as for separating the food from the background in theimage.

In an embodiment, a smartphone includes a back-facing camera, a 3Dimaging sensor, and a millimeter-wave radar. The 3D imaging sensor isimplemented with a ToF sensor, e.g., in combination with the back-facingcamera. The ToF sensor and the millimeter-wave radar, and optionally theback-facing camera, are used in conjunction to classify food based ontheir macronutrients composition. Other implementations are alsopossible. For example, some embodiments may implement the 3D imagingsensor for generating the 3D image using a Lidar of a phone.

FIG. 6 shows smartphone 600, according to an embodiment of the presentinvention. Smartphone 600 includes sensor module 602, touchscreen 604,and processor 606. Sensor module 602 includes a back-facing camera 712,a ToF sensor 714, and a millimeter-wave radar 716.

Some embodiments may be implemented in form factors different than asmartphone. For example, some embodiments may be implemented inhead-mounted computing devices, such as augmented reality glasses. Someembodiments may be implemented in kiosks in a cafeteria or restaurants,e.g., such as a restaurant with self-serving buffet bars. Someembodiments may be implemented in specialized consumer devices, such asa home appliance that analyzes food, such as a smart scale or smartmicrowave. Other implementations are possible.

As shown in e.g., FIG. 6 , some embodiments implement the 3D imagingsensor with a ToF sensor. Some embodiments implement the 3D imagingsensor in other ways. For example, some embodiments may use, instead ofor in addition to the ToF sensor, ultrasonic sensors, multiple RGBcameras, an array of radar sensors, and/or Lidar sensor to implement the3D imaging sensor.

In some embodiments, the radar may be substituted or augmented by usingother sensors, such as ultrasonic spectrometers, hyperspectral imagers,or NIR spectrometers. For example, in some embodiments, an ultrasonicsensor is used to visualize the internal composition of a food (e.g.,measure the thickness of a layer of fat on meat or detect the presenceof chunky food in soups). In some embodiments, a hyperspectral imager isused to detect impurities or spoilage.

Touchscreen 604, RGB camera 712, and ToF sensor 714 may be implementedin any way known in the art. In some embodiments, the combination of RGBcamera 712 and ToF sensor 714 may be operate as a 3D imaging sensor 711.

Processor 606 may be implemented in any way known in the art, such as ageneral purpose or custom processor, controller or digital signalprocessor (DSP) that includes, for example, combinatorial circuitscoupled to a memory. In some embodiments, processor 1120 may include anartificial intelligence (AI) accelerator.

In some embodiments, processor 606 is configured to run an operatingsystem of the smartphone 600, run apps, control peripherals (e.g., RGBcamera 712, ToF sensor 714 and millimeter-wave radar 716) and/or performother tasks associated with smartphone 600.

FIG. 7 shows a flow chart of embodiment method 700 for classifying food,according to an embodiment of the present invention. The description ofmethod 700 assumes an implementation with smartphone 600. It isunderstood that method 700 may be implemented in other types of devices.

During step 702, a user, such as a human, opens an app in smartphone600, the app having system-level permission to access one or moresensors of sensor module 602. During step 704, the software applicationdisplays food in screen 604 using, e.g., the back-facing RGB camera 712of smartphone 600.

During step 706, the user positions smartphone 600 so that the desiredportion of the food (e.g., the entire portion of food to be analyzed) isshown in screen 604.

During step 708, the user triggers analysis of the food displayed in thescreen 604, e.g., by touching a button of smartphone 600, such as byclicking in a virtual button displayed in touchscreen 604.

During step 710, the app uses one or more sensors of smartphone 600 toidentify the food displayed in the screen 604. For example, in someembodiments, the back-facing RGB camera 712 of smartphone 600 is used toidentify the food using conventional image recognition algorithms. Forexample, in some embodiments, a deep learning algorithm, such asAlexNet, ResNet, SqueezeNet, or DenseNet, may be used.

In some embodiments, the millimeter-wave radar 716 of smartphone 600 isused to identify the food. For example, in some embodiments,millimeter-wave radar 716 is used to generate 2D radar images. A neuralnetwork, such as a deep convolutional neural network (DCNN), may be usedin conjunction with radar signal processing (e.g., MTI filtering, rangeFFT and Doppler FFT) to classify the food.

In some embodiments, the ToF sensor 714 of smartphone 600 is used toidentify the food. For example, in some embodiments, depth distortiongenerated based on ToF data is used to distinguish between thebackground (e.g., table, plate, bowl, eating utensils, etc.) and thefood.

In some embodiments, more than one sensor (e.g., two or more of ToFsensor 714, millimeter-wave radar 716, or RGB camera 712) may becombined to identify the food. For example, in some embodiments, objectsin the field-of-view (FoV) of the sensors 712, 714, and 716 aresegmented based on data from ToF sensor 714, millimeter-wave radar 716,and RGB camera 712 to identify individual objects. For example, in anembodiment, an image of a burger may be segmented, e.g., into table,plate, top half of the hamburger bun, tomatoes, lettuce, cheese, meat,and bottom half of the hamburger bun, where the identified food includesthe top and bottom half of the hamburger bun, the tomatoes, the lettuce,the cheese, and meat, and the non-food includes the table and the plate.For example, in some embodiments, depth distortion (from ToF sensor 714)is used in relation to the image generated by RGB camera 712 to identifyobjects of different materials.

In some embodiments, the identification step 710 is based on food model718. Food model 718 may be implemented with, e.g., a neural network(e.g., such as a DCNN), to differentiate between different food andnon-food. The food model may be trained, e.g., during a characterizationstep (not shown), in which a training dataset that includes pre-labeleddata associated with food and non-food is used to optimize the foodmodel based on the output of one or more sensors (e.g., ToF sensor 714,millimeter-wave radar 716, and/or RGB camera 712).

In some embodiments, food model 718 may be stored in non-volatile memoryof smartphone 600.

In some embodiments, the image displayed on the screen 604 may bemodified as a result of the food identification step. For example, insome embodiments, portions of the image that are not food may be grayedout or dimmed. For example, FIG. 8 shows the screen 604 of smartphone600 while displaying a burger, in which areas of the screen 604 that arenot displaying food are grayed out, according to an embodiment of thepresent invention.

During step 720, the user may manually correct the results of the foodidentification step (step 710). For example, in some embodiments, theuser may click on the touchscreen 604 to associate a portion of thescreen 604 to food or non-food. In some embodiments, the characteristicsof corrected objects may be used to further refine the food model.

During step 722, ToF sensor 714 is used to identify the volume of theidentified food (e.g., identified during step 710), e.g., by generatinga depth map and associating it with the segmented image. For example, a2D image generated by RGB camera 712 may be associated with ToF data(e.g., a depth map) generated by ToF sensor 714 to generate a 3D image.The volume may be identified from the 3D image.

In some embodiments, the 3D image is processed separately, as a 2D imagefrom RGB camera 712 and a depth map from ToF sensor 714 without creatinga 3D image file (e.g., by associating points of the depth map withcorresponding points of the 2D image without combining the 2D image andthe depth map into the same file). For example, in some embodiments, thevolume of the food is identified from the 3D image by segmenting the 2Dimage, associating points of the depth map to the segmented image, anddetermining the volume of the food based on the ToF data associated withthe segmented image (e.g., without creating a 3D image file).

In some embodiments, the 2D image from RGB camera 712 and the depth mapfrom ToF sensor are captured simultaneously. In some embodiments, the 2Dimage from RGB camera 712 and the depth map from ToF sensor are capturedat different times. In some embodiments, the 3D image includes themerged data from the 2D image from RGB camera 712 and from the depth mapfrom ToF sensor 714. In some embodiments, the 3D image is rendered intouchscreen 604. In some embodiments, the 3D image is used withoutdisplaying the 3D image in a screen.

During step 724, the composition of the identified food is determined.For example, in some embodiments, RGB camera 712 and/or ToF data fromToF sensor 714 is used to classify the identified food (e.g., an apple).The application then consults nutrition database 726 (e.g., in the cloudand/or stored locally in smartphone 600 in an app database) to infer the(generic) composition (e.g., the macronutrients) of the classified food.

In some embodiments, during step 724, millimeter-wave radar 716 extractsa food composition signature of the food (e.g., a measurement vector y,such as explained in more detailed below, e.g., with respect to FIGS.13A and 13B and Equations 7 and 10) and compares such signature withradar model database 728 to identify a specific composition of theidentified food. For example, in some embodiments, radar model database728 holds models for each type of classified food (e.g., potato, pear,corn, etc.). In some embodiments, for each type of classified food, aradar signature has been computed for known compositions. For example, amodel for apples holds signatures for apples of different ripeness andsugar levels. Similarly, a model for french fries holds signatures forfrench fries of varying fat levels and potato types. Once the particulartype of food has been identified (e.g., an apple of a particularripeness and sugar level), nutrition database 726 may be consulted toidentify the specific composition (e.g., macronutrients) of theidentified food.

In some embodiments, classification of different types of food displayedon the screen 604 (e.g., different types of food on a plate) isperformed (e.g., sequentially or simultaneously). For example, a platefilled with pasta on a first half, and salad on a second half may firstclassify the pasta (e.g., by consulting databases 726 and 728 based onthe output of sensors 712, 714 and 716 associated to the region of thescreen 604 displaying the pasta), and then the salad (e.g., byconsulting databases 726 and 728 based on the output of sensors 712, 714and 716 associated to the region of the screen 604 displaying thesalad).

In some embodiments, databases 726 and 728 may be stored in non-volatilememory of smartphone 600.

During step 730, the composition of the food estimated during step 724is displayed. For example, FIGS. 9 and 10 show examples of possible waysto display the composition of the food estimated during step 724,according to embodiments of the present invention.

As shown in FIG. 9 , in some embodiments, the composition of may includea macronutrient composition of the food.

As shown in FIG. 10 , in some embodiments, the composition of the foodestimated during step 724 may be recorded and tracked, and may bedisplayed, e.g., by day, week, or month, or by a custom timeframe, forexample. In some embodiments, the food composition history may be storedin a memory of smartphone 600. In some embodiments, the history of foodcomposition intake may be used, e.g., as the basis to provide dietarysuggestions.

In some embodiments, the smartphone app may provide suggestions for theuser's next meal, e.g., to balance their macronutrient profile or meettheir dietary goals, such as eating an extra serving of vegetables orprotein.

In some embodiments, the smartphone app tracks the user's diet and maycorrelate diet with other information, such as weight, blood sugar,cholesterol, exercise/activity level, sleep, or pain level, e.g.,depending on the user's goals.

In some embodiments, the smartphone app may suggest modifications to theuser's dietary goals.

Some embodiments may advantageously be used for treatments and/orprevention of obesity, heart disease, cancer, diabetes, and/or otherchronic health problems. Some embodiments may be used to promotewellness (e.g., weight loss),In some embodiments, material componentidentification is performed by a millimeter-wave radar (e.g., 716)configured to perform minimum variance distortionless response (MVDR),also known as Capon, beamforming. For example, Figure ii shows aschematic diagram of millimeter-wave radar 716, according to anembodiment of the present invention.

During normal operation, millimeter-wave radar sensor 1102 operates as afrequency-modulated continuous-wave (FMCW) radar sensor and transmits aplurality of radar signals 1106, such as chirps, towards scene 1130using transmitter (TX) antenna 1114. The radar signals 1106 aregenerated using RF and analog circuits 1104. The radar signals 1106 maybe in the 20 GHz to 122 GHz range.

The objects in scene 1130 may include food, such as single items offood, such as a carrot or an apple, combination of food items, such as aburger, and non-food items, such as plates, silverware, and napkins, forexample. Other objects may also be present in scene 1130, such as walls,countertops, furniture, etc.

The transmitted radar signals 1106 are reflected by objects in scene1130. The reflected radar signals 1108, which are also referred to asthe echo signal, are received by receiver (RX) antenna 1116. RF andanalog circuits 1104 processes the received reflected radar signals 1108using, e.g., band-pass filters (BPFs), low-pass filters (LPFs), mixers,low-noise amplifier (LNA), and/or intermediate frequency (IF) amplifiersin ways known in the art to generate an analog signal x_(out)(t).

The analog signal x_(out)(t) is converted to raw digital datax_(out_dig)(n) using ADC 1112. The raw digital data x_(out_dig)(n) isprocessed by processor 1120 to identify and classify the food componentspresent in scene 1130.

Controller 1110 controls one or more circuits of millimeter-wave radarsensor 1102, such as RF and analog circuit 1104 and/or ADC 1112.Controller 1110 may be implemented as a custom or general purposescontroller or processor, that includes, e.g., combinatorial circuitscoupled to a memory.

Processor 1120 may be implemented in any way known in the art, such as ageneral purpose or custom processor, controller or digital signalprocessor (DSP) that includes, for example, combinatorial circuitscoupled to a memory. In some embodiments, processor 1120 may include anartificial intelligence (AI) accelerator. In some embodiments, processor1120 may be implement a portion of controller 1110. In some embodiments,a portion of processor 1120 may be implemented by processor 606.

RF and analog circuits 1104 may be implemented, e.g., as shown in FIG.11 . During normal operation, VCO 1136 generates a linear frequencychirp (e.g., from 57 GHz to 64 GHz), which is transmitted bytransmitting antenna 1114. The VCO 1136 is controlled by PLL 1134, whichreceives a reference clock signal (e.g., 80 MHz) from referenceoscillator 1132. PLL 1134 is controlled by a loop that includesfrequency divider 1138 and amplifier 1140.

The linear chirp transmitted by transmitting antenna 1114 is reflectedby objects in scene 1130 and received by receiving antenna 1116. Theecho received by transmitting antenna 1116 is mixed with a replica ofthe signal transmitted by transmitting antenna 1114 using mixer 1146 toreduce an intermediate frequency (IF) signal x_(IF)(t) (also known asthe beat signal). In some embodiments, the beat signal x_(IF)(t) has abandwidth between 10 kHz and 1 MHz. A beat signal x_(IF)(t) with abandwidth lower than 10 kHz or higher than 1 MHz is also possible.

The beat signal x_(IF)(t) is filtered with low-pass filter (LPF) 1148and then sampled by ADC 1112. ADC 1112 is advantageously capable ofsampling the filtered beat signal x_(out)(t) with a sampling frequencythat is much smaller than the frequency of the signal received byreceiving antenna 1116. Using FMCW radars, therefore, advantageouslyallows for a compact and low cost implementation of ADC 1112, in someembodiments.

The raw digital data x_(out_dig)(n), which in some embodiments is thedigitized version of the filtered beat signal x_(out)(t), is (e.g.,temporarily) stored (e.g., in matrices of N_(c)×N_(s), where N_(c) isthe number of chirps considered in a frame and N_(s) is the number oftransmit samples per chirp) for further processing.

In some embodiments, ADC 1112 is a 12-bit ADC. ADCs with higherresolution, such as 14-bits or higher, or with lower resolution, such as10-bits, or lower, may also be used.

FIG. 12 shows a block diagram of embodiment method 1200 for performingfood classification (step 724) using millimeter-wave radar 716,according to an embodiment of the present invention.

During step 1202, raw ADC data x_(out_dig)(n) is received. The raw ADCdata x_(out_dig)(n) includes separate baseband radar data from multipleantennas 1116. During step 1202, signal conditioning, low pass filteringand background removal is performed on the raw ADC data x_(out_dig)(n).For example, in some embodiments, the raw ADC data x_(out_dig)(n) arefiltered, DC components are removed to, e.g., remove the Tx-Rxself-interference and optionally pre-filtering the interference colorednoise. In some embodiments, filtering includes removing data outliersthat have significantly different values from other neighboringrange-gate measurements. Thus, this filtering also serves to removebackground noise from the radar data. For example, in some embodiments,a Hampel filter is applied with a sliding window at each range-gate toremove such outliers. Alternatively, other filtering for rangepreprocessing known in the art may be used.

During step 1204, a series of FFTs are performed on conditioned radardata X_(conditioned)(n) produced during step 1202. In some embodiments,a windowed FFT having a length of the chirp (e.g., 256 samples) iscalculated along each waveform for each of a predetermined number ofchirps in a frame of data. Alternatively, other frame lengths may beused. The FFTs of each waveform or chirp may be referred to as a “rangeFFT.” In alternative embodiments, other transform types could be usedbesides an FFT, such as a Discrete Fourier Transform (DFT) or az-transform.

During step 1206, capon beamforming is performed. A beam is formed atthe transmitter by post processing a plurality of baseband signals basedon a plurality of signals received by different receivers (e.g.,different RX antennas 1116) or a combination thereof. Implementingbeamforming by post processing received baseband signals may allow forthe implementation of a low complexity transmitter.

In some embodiments, millimeter-wave radar 716 includes N_(t)=2 transmit(TX) elements and N_(r)=2 receive (RX) elements arranged in a array.Accordingly, there are N_(t)×N_(r)=4 distinct propagation channels fromthe TX array to the RX array in an array configuration for azimuth angleprofiling. If the transmitting source (TX channel) of the receivedsignals can be identified at the RX array, a virtual phased array ofN_(t)×N_(r) elements can be synthesized with N_(t)+N_(r) antennaelements. In various embodiments, a time division multiplexed MIMO arrayprovides a low cost solution to a fully populated antenna aperturecapable of near field imaging.

The transmitter steering vector may be expressed as

$\begin{matrix}{{a_{m}^{Tx}\left( \theta \right)} = e^{{- j}2{\pi \cdot \frac{d_{m}^{Tx} \cdot {\sin(\theta)}}{\lambda}}}} & (1)\end{matrix}$

where θ is the angle of the target (e.g., the angle pointing towards thelocation where the food), λ is the wavelength of the transmitted signal(e.g., the center frequency of the chirp), d_(m) ^(Tx) is the positionalvector of the m^(th) transmitting antenna with respect to the center ofthe transmitting antenna array, and m=1, 2 (for each of the TXantennas).

The receiver steering vector may be expressed as

$\begin{matrix}{{a_{m}^{Rx}\left( \theta \right)} = e^{{- j}2{\pi \cdot \frac{d_{m}^{Rx} \cdot {\sin(\theta)}}{\lambda}}}} & (2)\end{matrix}$

where d_(n) ^(Rx) is the positional vector of n^(th) receiving antennawith respect to the center of the receiving antenna array, and n=1, 2(for each of the RX antennas).

In some embodiments, the azimuth imaging profile for range bin l isgenerated using the Capon spectrum from the beamformer. The Caponbeamformer is computed by minimizing the variance/power of noise whilemaintaining a distortion-less response towards a desired angle. Thecorresponding quadratic optimization problem can be expressed as

min_(ω)ω^(H) Cωs.t. ω^(H)(α^(Tx)(θ)⊕α^(Rx)(θ))=1  (3)

where ω represent the beamforming weights for the particular angle θ(where the constraint function is ensuring unit response), and C is thecovariance matrix of noise.

Equation 3 is an optimization that has a closed form expression that maybe given by

$\begin{matrix}{w_{capon} = {C^{- 1} \cdot {\frac{a(\theta)}{{a^{H}(\theta)}C^{- 1}{a(\theta)}}.}}} & (4)\end{matrix}$

The spatial spectrum, thus, may be given by

$\begin{matrix}{{P_{l}(\theta)} = \frac{1}{\left( {{a^{Tx}(\theta)} \otimes {a^{Rx}(\theta)}} \right)^{H}{C_{l}^{- 1}\left( {{a^{Tx}(\theta)} \otimes {a^{Rx}(\theta)}} \right)}}} & (5)\end{matrix}$

where l=o, . . . , L, L being the total number of range bins.

In some embodiments, C_(l) is estimated instead of the noise covariancematrix C. C_(l) is the estimated covariance matrix based on the rangedata at range bin l (e.g., if the virtual antenna has a dimension 8,C_(l) is a matrix is 8×8 for each range bin). C_(l) includes the signalcomponent (similarly than noise covariance matrix C) and can beestimated using sample matrix inversion (SMI) technique. In someembodiments, estimating C_(l) may be simpler than estimating the noisecovariance matrix C. C_(l) may be estimated by

$\begin{matrix}{C_{l} = {\frac{1}{N}{\sum\limits_{k = 1}^{K}{{s_{l}^{IF}(k)}{s_{l}^{IF}(k)}^{H}}}}} & (6)\end{matrix}$

where K represents the number of snapshots, and s_(l) ^(IF)(k)represents the received signal at l^(th) range bin at k^(th)measurement. In some embodiments, the snapshot could represent the chirpnumber or frame number based on which data is used to update thecovariance matrix.

During step 1208, the discrete cosine transform (DCT) is performed. Insome embodiments, step 1208 advantageously allows for achievingdimensionality reduction. In some embodiments, step 1208 may be omitted.

During step 1210, sparse decomposition (with dictionary learning) isperformed. Step 1210 may be understood in view of FIGS. 13A and 13B.FIGS. 13A and 13B illustrate possible embodiments for performing step1210, according to an embodiment of the present invention. In someembodiments, the representation illustrated in FIG. 13B corresponds toperforming step 1210 directly from the output of step 1206 (omittingstep 1208).

As shown in FIG. 13A, the sensing equation may be given by

y=Ax  (7)

where y is the measurement vector (e.g., measurements performed bymillimeter-wave radar 716, such as measurements at the output of step1208), A is the sensing matrix, where A has dimensions of m×n, (whereeach column of A corresponds to a radar signature of a type of food) andx is a vector including the food composition to be identified (e.g.,where each cell (row) of the vector x corresponds to a particular foodtype (column of matrix A)). In some embodiments, matrix A may be storedin radar model database 728.

Since there is only one measurement vector y and there are severalunknown components (food composition) to be identified, the problem isunder-determined, which in a normal formulation entails infinitesolutions. To avoid infinite solutions, the structure is forced to havea dictionary A, so each column of A represents a component and x iseither 0 or 1, indicating whether that component is present in themeasurement vector y or not. For example, matrix A may be understood asa learned dictionary, in which each column corresponds to a signature ofa type of food (e.g., potato, pear, corn, etc.) and each row of eachcolumn (e.g., each cell of the column) corresponds to a range-azimuth(and optionally angle) collapsed into a single vector (e.g., as a resultof performing a vectorization operation).

Matrix A may be generated by characterizing hundreds or thousands ofdifferent types of foods (e.g., potato, pear, corn, etc.). For example,in some embodiments, matrix A may be generated by collecting radarmeasurements (e.g., such as measurements at the output of step 1208 orstep 1206) of different types of foods (e.g., potato, pear, corn, etc.),where the measurements of each type of food is arranged in respectivecolumns of matrix A.

Various approximation (also referred to as pursuit) algorithms may beused to address the sparse representation problem. For example, in someembodiments,

$\begin{matrix}\begin{matrix}{\min\limits_{s \in R^{p}}{❘s❘}_{0}} & {{s.t.{❘{y - {As}}❘}_{2}^{2}} \leq \varepsilon}\end{matrix} & (8)\end{matrix}$

where p represents the number of components in the sensed image, |·|₀denotes a L0 norm.

In some embodiments, since the problem is NP-Hard, it may beapproximated as

$\begin{matrix}{{\min\limits_{s \in R^{p}}{❘s❘}_{1}} + {\lambda{❘{y - {As}}❘}_{2}^{2}}} & (9)\end{matrix}$

where |·|₁ denotes a L1 norm and |·|₂ denotes a L2 norm.

In some embodiments, matching pursuit (MP) is used to solve theoptimization problem of Equation 9. Matching pursuit may be understoodas a greedy iterative algorithm that gradually finds the locations ofthe non-zeros in x one at a time. In each step, the MP algorithm findsthe column in A that best correlates with the current residual, and thenupdates this residual to take the new component and its coefficient intoaccount. Matching pursuit might pick the same atom multiple times.

In some embodiments, orthogonal matching pursuit is used to solve theoptimization problem of Equation 9. Orthogonal matching pursuit operatesin a similar manner as MP. In orthogonal matching pursuit, however, allthe non-zero coefficients are updated by a Least-Squares in each of thealgorithm's step. As a consequence, the residual is orthogonal to thealready chosen component, and thus an atom cannot be picked more thanonce.

After solving Equation 7, e.g., using optimization techniques, vector xis obtained. In some embodiments, each cell of vector x corresponds to aparticular types of foods (e.g., potato, pear, corn, etc.) and includesa number between 0 and 1, such as 0, or 1. An additional thresholdingstep (e.g., rounding) may be performed to obtain a vector x with only 0and 1. Cells of vector x having 1 are indicative of the correspondingfood being present in the measured food (as reflected by vector y) whilecells having a 0 are indicative of the corresponding food not beingpresent in the measured food. In other words, in some embodiments, aftersolving for x in Equation 7, the combination of foods (columns of matrixA) that better approximate the measurement vector y are indicated by thecells labeled as 1 in vector x. Thus, in some embodiments performingstep 1210 in accordance with the embodiment of FIG. 13A, the materialcomponent classification obtained as the output of step 1210 may be aclassification of types of food (e.g., potato, pear, corn, etc.).

As shown in FIG. 13B, the sensing equation may be given by

y=Φψs  (10)

where y is the measurement vector (e.g., measurements performed bymillimeter-wave radar 716, such as measurements at the output of step1206), ψ is a first sensing matrix (e.g., similar or identical tosensing matrix A, Φ is a second sensing matrix, and s is a vectorincluding the food composition to be identified. In some embodiments,matrices ψ and Φ may be stored in radar model database 728.

In some embodiments in which step 1208 is omitted, Matrix Φ may beunderstood as a learned dictionary in which each column corresponds to asignature of a combination of 2 or more foods (e.g., 30% potato, 45%pear, and 25% corn) and each row of each column (e.g., each cell of thecolumn) corresponds to a range-azimuth (and optionally angle) collapsedinto a single vector (e.g., as a result of performing a vectorizationoperation).

Equation 10 may be solved for s using optimization techniques (e.g.,matching pursuit). In some embodiments, the resulting vector s includesat most K non-zero cells (K-sparse), where K is a positive integergreater than 1, such as 8, 12, 13, or more. Each of the cells of vectors corresponds to a particular type of food (e.g., potato, pear, corn,etc.), and the value of each cell of s is indicative of the proportionof the associated type of food in the measurement vector y. For example,vector s may have, after solving Equation 10 for a particularmeasurement vector y, 3 non-zero cells corresponding to 3 types of foods(e.g., potato, pear, and corn), and the values of such 3 non-zero cells(e.g., 0.3, 0.45, 0.25) are indicative of the proportion of such foodsin the measurement vector y (e.g., 30% potato, 45% pear, and 25% corn).Thus, in some embodiments performing step 1210 in accordance with theembodiment of FIG. 13B, the material component classification obtainedas the output of step 1210 may be a classification of types andproportion of food (e.g., 30% potato, 45% pear, and 25% corn).

In some embodiments, an additional step may be performed after obtainingthe types and proportion of foods to obtain the proportion ofmacronutrients associated with the measurement vector y (e.g.,associated with the food being analyzed with the app in smartphone 600(e.g., as illustrated in FIG. 8 ). For example, after determining thatthe food being analyzed includes (e.g., 30% potato, 45% pear, and 25%corn), a database (e.g., nutrition database 728) may be consulted toobtain the macronutrients (e.g., carbohydrates, proteins, fats, etc.) ofthe identified foods (e.g., potato, pear, and corn), and suchmacronutrients being multiplied by their proportions to arrive at acomposition of macronutrients of the analyzed food). Such composition ofmacronutrients may be displayed, e.g., as illustrated in FIGS. 9 and/or10 , for example. Thus, in some embodiments performing step 1210 inaccordance with the embodiment of FIG. 13B, the material componentclassification obtained as the output of step 1210 may be aclassification of types and proportion of macronutrients (e.g., 30%carbohydrates, 45% protein, and 25% fat).

In some embodiments, the proportion of macronutrients obtained from theoutput of step 1210 may be combined with the volume of the food (e.g.,identified during step 722) to determine the quantity of macronutrients.

FIG. 14 shows a block diagram of embodiment method 1400 for performingfood classification (step 724) using millimeter-wave radar 716,according to an embodiment of the present invention. Method 1400implements steps 1202, 1204, and 1206 in a similar manner as method1200.

During step 1402, a DCNN is used to determine the composition of foodbased on the output from step 1206. The output of step 1402 may be,e.g., types of food present in the identified food (e.g., potato, pear,corn, etc.), types and proportion of food (e.g., 30% potato, 45% pear,and 25% corn) and/or types and proportion of macronutrients (e.g., 30%carbohydrates, 45% protein, and 25% fat).

In some embodiments, the DCNN used during step 1402 may be implementedwith 5 residual convolutional layers with maxpooling, followed by 2fully-connected layers, followed by a softmax output. Otherimplementations are also possible.

In some embodiments, the DCNN used during step 1402 may be trained usingsupervised training based on training data that includes sample foodsand combination of foods. FIG. 15 illustrates a block diagram showingtraining and operation of a DCNN used during step 1402 foridentification of food components, according to an embodiment of thepresent invention. The top portion 1500 of FIG. 15 illustrates thetraining of the DCNN, while the bottom portion 1520 of FIG. 15illustrates the trained DCNN during normal operation.

During training, radar ADC data 1502, e.g., from ADC 1112, (alsoreferred to as training data 1502), is transformed into feature vectors1510 and corresponding component labels 1512. Feature vectors 1510represent sets of generated vectors that are representative of trainingdata 1502. The feature vectors 1510 are generated by a data preparationstep 1504 (e.g., steps 1202 and 1204), and a range-angle image (RAI)formation step (e.g., step 1206).

Component labels 1512 represent user metadata associated with thecorresponding training data 1502 and feature vectors 1510 (andcorresponding to the dictionary vectors in step 1210, such as inmatrices A, ψ and Φ). Component labels 1512 are generated by a dataannotation step 1508, in which metadata is generated to identify thefood components present in the training data set. For example, if asample of training data 1502 corresponds to a particular carrot, thecomponent label associated with the corresponding feature vectorincludes the components of such particular carrot.

The featured vectors 1510 and corresponding component labels 1512 arefed into the DCNN optimization algorithm 1514. The DCNN optimizationalgorithm 1514 optimizes the weights of the DCNN to minimize themulti-label loss.

During the optimization process, multiple samples, such as thousands ofsamples are used to train the DCNN. The trained DCNN is the predictionmodel 1530 used to predict food components during normal operation.

The training performance of the DCNN optimization algorithm 1514 may bedetermined by calculating the cross-entropy performance. In someembodiments, the DCNN optimization algorithm 1514 iteratively adjustsimage formation parameters, such as the number of snapshots, for aclassification accuracy of at least 90%. Alternatively, otherclassification accuracies could be used.

In some embodiments, an RMSprop optimizer is used to optimize the DCNN.Other optimizers, such as a gradient decent optimizer, and gradientdecent with momentum optimizer, ADAM, may also be used. In someembodiments, the learning rate is l_(r) is 0.0001, with ρ of 0.9 and ϵof 10⁻⁸. Other learning parameter values may also be used.

During normal operation, radar ADC data 1522, e.g., from ADC 1112, isprepared during step 1524 (e.g., steps 1202 and 1204), and featuredvectors 1528 are generated during a RAI formation step (e.g., step1206). Featured vectors 1528 are fed to the DCNN (prediction model) 1530to generate the predicted composition of the food.

Example embodiments of the present invention are summarized here. Otherembodiments can also be understood from the entirety of thespecification and the claims filed herein.

EXAMPLE 1

A method for estimating a composition of food, the method including:receiving a first 3D image; identifying food in the first 3D image;determining a volume of the identified food based on the first 3D image;and estimating a composition of the identified food using amillimeter-wave radar.

EXAMPLE 2

The method of example 1, further including: receiving data from a 3Dimaging sensor; and generating the first 3D image based on the data fromthe 3D imaging sensor.

EXAMPLE 3

The method of one of examples 1 or 2, where a mobile device includes the3D imaging sensor, the millimeter-wave radar, and a screen, the methodfurther including: positioning the mobile device to display the food onthe screen; and triggering food analysis after displaying the food onthe screen, and before receiving the data from the 3D imaging sensor.

EXAMPLE 4

The method of one of examples 1 to 3, further including opening an appin the mobile device before triggering the food analysis.

EXAMPLE 5

The method of one of examples 1 to 4, further including: displaying asecond image on a screen, the second image based on the first 3D image;and manually adjusting which portion of the screen corresponds to foodbased on a user input.

EXAMPLE 6

The method of one of examples 1 to 5, where identifying the foodincludes using a food model, the method further including updating thefood model based on the manually adjusting.

EXAMPLE 7

The method of one of examples 1 to 6, further including displaying asecond image on a screen, the second image based on the first 3D image,where displaying the second image on the screen includes graying outportions of the second image that correspond to non-food.

EXAMPLE 8

The method of one of examples 1 to 7, further including displaying anestimated composition of the identified food on a screen.

EXAMPLE 9

The method of one of examples 1 to 8, further including recording ahistory of food composition intake based on estimated food compositions;and displaying dietary suggestions based on the history of foodcomposition intake.

EXAMPLE 10

The method of one of examples 1 to 9, where identifying the food in thefirst 3D image includes segmenting the first 3D image based on outputsfrom the 3D imaging sensor and the millimeter-wave radar.

EXAMPLE 11

The method of one of examples 1 to 10, where identifying food in thefirst 3D image includes using an image recognition algorithms.

EXAMPLE 12

The method of one of examples 1 to 11, further including generating a 2Dradar image with the millimeter-wave radar, where identifying food inthe first 3D image includes using the 2D radar image.

EXAMPLE 13

The method of one of examples 1 to 12, where estimating the compositionof food includes: generating a food composition signature of the foodusing the millimeter-wave radar; providing a first classification of thefood by comparing the food composition signature to a radar modeldatabase; and estimating the composition of the food based on the firstclassification and a nutrition database.

EXAMPLE 14

The method of one of examples 1 to 13, where estimating the compositionof food includes: receiving radar data from an ADC of themillimeter-wave radar; preprocessing the received radar data to generatea radar image; and using a deep convolutional neural network (DCNN) toestimate the composition of the food based on the radar image.

EXAMPLE 15

The method of one of examples 1 to 14, where the radar image is arange-angle image (RAI).

EXAMPLE 16

The method of one of examples 1 to 15, further including training theDCNN.

EXAMPLE 17

The method of one of examples 1 to 16, where estimating the compositionof food includes: receiving radar data from an ADC of themillimeter-wave radar; preprocessing the received radar data to generatea radar image; performing a discrete cosine transform on the radar imageto generate a transformed output; and performing a sparse decompositionbased on the transformed output to estimate the composition of the food.

EXAMPLE 18

A mobile device including: a 3D imaging sensor configured to generate afirst 3D image; a millimeter-wave radar configured to transmit radarsignals and receive reflected radar signals; and a processor configuredto: identify food in the first 3D image; determine a volume of theidentified food based on the first 3D image; and estimate a compositionof the identified food based on the reflected radar signals.

EXAMPLE 19

The mobile device of example 18, where the 3D imaging sensor includes atime-of-flight sensor.

EXAMPLE 20

The mobile device of one of examples 18 or 19, where the 3D imagingsensor further includes an RGB camera.

EXAMPLE 21

The mobile device of one of examples 18 to 20, where the 3D imagingsensor includes a Lidar sensor.

EXAMPLE 22

The mobile device of one of examples 18 to 21, where the mobile deviceis a smartphone.

EXAMPLE 23

The mobile device of one of examples 18 to 22, further including ascreen configured to display a second image that is based on the first3D image.

EXAMPLE 24

The mobile device of one of examples 18 to 23, where the processor isconfigured to determine a quantity of macronutrients of the identifiedfood based on the determined volume and the estimated composition of theidentified food.

EXAMPLE 25

A system including: a time-of-flight sensor; a camera; and amillimeter-wave radar configured to transmit radar signals and receivereflected radar signals; and a processor configured to: identify foodbased on an output of the camera; determine a volume of the identifiedfood based on an output of the time-of-flight sensor; estimate acomposition of the identified food based on the reflected radar signals;and determine a quantity of macronutrients of the identified food basedon the determined volume and the estimated composition of the identifiedfood.

EXAMPLE 26

The system of example 25, where the processor is further configured togenerate a measurement vector based on the reflected radar signals,where estimating the composition of the identified food includesestimating the composition of the identified food based on themeasurement vector and a first sensing matrix, the first sensing matrixincluding radar signatures of a plurality of types of food.

EXAMPLE 27

The system of one of examples 25 or 26, where estimating the compositionof the identified food is further based on a second sensing matrix, thesecond sensing matrix including radar signatures of combinations of 2 ormore types of foods.

EXAMPLE 28

A method for estimating a composition of food, the method including:receiving radar data from an ADC of a millimeter-wave radar;preprocessing the received radar data to generate a radar image;generating a measurement vector based on the radar image; and estimatinga composition of food based on the measurement vector and a firstsensing matrix, the first sensing matrix including radar signatures of aplurality of types of food.

EXAMPLE 29

The method of example 28, where the estimated composition of foodincludes one or more types of food.

EXAMPLE 30

The method of one of examples 28 or 29, where estimating the compositionof food is further based on a second sensing matrix, the second sensingmatrix including radar signatures of combinations of 2 or more types offoods.

EXAMPLE 31

The method of one of examples 28 to 30, where the estimated compositionof food includes proportions of fats, proteins, and carbohydrates.

While this invention has been described with reference to illustrativeembodiments, this description is not intended to be construed in alimiting sense. Various modifications and combinations of theillustrative embodiments, as well as other embodiments of the invention,will be apparent to persons skilled in the art upon reference to thedescription. It is therefore intended that the appended claims encompassany such modifications or embodiments.

What is claimed is:
 1. A method for estimating a composition of asubstance, the method comprising: receiving a first 3D image;identifying the substance in the first 3D image; determining a volume ofthe identified substance based on the first 3D image; and estimating thecomposition of the identified substance using a millimeter-wave radar,wherein estimating the composition of the identified substancecomprises: receiving radar data from an ADC of the millimeter-waveradar, preprocessing the received radar data to generate a radar image,and using a deep convolutional neural network (DCNN) to estimate thecomposition of the substance based on the radar image.
 2. The method ofclaim 1, further comprising: receiving data from a 3D imaging sensor;and generating the first 3D image based on the data from the 3D imagingsensor.
 3. The method of claim 2, wherein a mobile device comprises the3D imaging sensor, the millimeter-wave radar, and a screen, the methodfurther comprising: positioning the mobile device to display thesubstance on the screen; and triggering a substance analysis afterdisplaying the substance on the screen, and before receiving the datafrom the 3D imaging sensor.
 4. The method of claim 3, further comprisingopening an app in the mobile device before triggering the substanceanalysis.
 5. The method of claim 2, wherein identifying the substance inthe first 3D image comprises segmenting the first 3D image based onoutputs from the 3D imaging sensor and the millimeter-wave radar.
 6. Themethod of claim 1, further comprising: displaying a second image on ascreen, the second image based on the first 3D image; and manuallyadjusting which portion of the screen corresponds to the substance basedon a user input.
 7. The method of claim 6, wherein identifying thesubstance comprises using a substance model, the method furthercomprising updating the substance model based on the manually adjusting.8. The method of claim 1, further comprising displaying an estimatedcomposition of the identified substance on a screen.
 9. The method ofclaim 1, wherein identifying the substance in the first 3D imagecomprises using an image recognition algorithms.
 10. The method of claim1, further comprising generating a 2D radar image with themillimeter-wave radar, wherein identifying the substance in the first 3Dimage comprises using the 2D radar image.
 11. The method of claim 1,wherein estimating the composition of the substance comprises:generating a composition signature of the substance using themillimeter-wave radar; providing a first classification of the substanceby comparing the composition signature to a radar model database; andestimating the composition of the substance based on the firstclassification and a nutrition database.
 12. The method of claim 1,wherein the radar image is a range-angle image (RAI).
 13. The method ofclaim 1, further comprising training the DCNN.
 14. A method forestimating a composition of a substance, the method comprising:receiving a first 3D image; identifying the substance in the first 3Dimage; determining a volume of the identified substance based on thefirst 3D image; and estimating the composition of the identifiedsubstance using a millimeter-wave radar, wherein estimating thecomposition of the identified substance comprises: receiving radar datafrom an ADC of the millimeter-wave radar, preprocessing the receivedradar data to generate a radar image, performing a discrete cosinetransform on the radar image to generate a transformed output, andperforming a sparse decomposition based on the transformed output toestimate the composition of the substance.
 15. A mobile devicecomprising: a 3D imaging sensor configured to generate a first 3D image;a millimeter-wave radar configured to transmit radar signals and receivereflected radar signals; and and a processor configured to: identify asubstance in the first 3D image; determine a volume of the identifiedsubstance based on the first 3D image; and estimate a composition of theidentified substance based on the reflected radar signals, by receivingradar data from an ADC of the millimeter-wave radar, preprocessing thereceived radar data to generate a radar image, and using a deepconvolutional neural network (DCNN) to estimate the composition of thesubstance based on the radar image.
 16. The mobile device of claim 15,wherein the 3D imaging sensor comprises a time-of-flight sensor.
 17. Themobile device of claim 16, wherein the 3D imaging sensor furthercomprises an RGB camera.
 18. The mobile device of claim 15, wherein the3D imaging sensor comprises a Lidar sensor.
 19. The mobile device ofclaim 15, wherein the mobile device is a smartphone.
 20. The mobiledevice of claim 15, further comprising a screen configured to display asecond image that is based on the first 3D image.
 21. A method forestimating a composition of a substance, the method comprising:receiving radar data from an ADC of a millimeter-wave radar;preprocessing the received radar data to generate a radar image;generating a measurement vector based on the radar image; and estimatingthe composition of the substance based on the measurement vector and apredetermined first sensing matrix, the first sensing matrix comprisingradar signatures of a plurality of types of substances wherein eachcolumn of the first sensing matrix comprises a radar signature thatcorresponds to a particular type of substance.
 22. The method of claim21, wherein the estimated composition of the substance comprises one ormore types of substances.
 23. The method of claim 21, wherein estimatingthe composition of the substance is further based on a second sensingmatrix, the second sensing matrix comprising radar signatures ofcombinations of two or more types of substances.
 24. The method of claim21, wherein estimating the composition of the substance furthercomprises: performing a discrete cosine transform on the radar image togenerate the measurement vector; and performing a sparse decompositionbased on the measurement vector to estimate the composition of thesubstance.